Impact of VR on Learning Experience compared to a Paper based Approach

ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal
Regular Issue, Vol. 12 N. 1 (2023), e31134
eISSN: 2255-2863
DOI: https://doi.org/10.14201/adcaij.31134

Different learning theories encourage different kinds of learning approaches. Following constructivist theories, learning experiences should be realistic in order to facilitate learning. Virtual Reality (VR) serious games could be a realistic learning approach without the challenges of the real situation. The serious game InGo allows a user to learn the intralogistics process of receiving goods. In this work we explore whether learning in VR is more effective concerning learning success and learning experience than traditional learning approaches. No significant difference between the two approaches concerning learning success is found. However, other factors that have a long term effect on learning, such as intrinsic motivation, flow and mood, are significantly higher for the VR approach. Thus, our research fits with past research which indicated the high potential of VR based learnig and educational games. This work encourages future research to compare VR based and traditional learning approaches in the long term.

While there has been an interest in Virtual Reality (VR) for a long time, with the simulation of reality being a topic as early as the 1930’s, recently there has been a strong increase in the interest in VR and especially in its use in education (Scavarelli et al., 2021). Technologically, VR can be defined as "an artificial environment which is experienced through sensory stimuli (such as sights and sounds) provided by a computer and in which one’s actions partially determine what happens in the environment" (Jerald, 2015). With the spread of VR and AR, there also has been a strong increase in their use in educational applications, especially in educational serious games (Scavarelli et al., 2021). Serious games are designed for a goal other than pure entertainment, often being educational or instructional (Schlüter and Kretschmer, 2020).

Previous work indicates positive effects on the learning experience of serious gaming and its outcomes, especially when comparing different performance parameters in the simulated real setting (Clark et al., 2016). The higher the virtuality of a learning experience, the deeper the participants immerse in the simulated working environment and thus they are less distracted (Lamb, 2016). This leads to the assumption that a VR based serious game might have further positive effects on the learning success, especially in a practice-dominated area like intralogistics processes. However, other aspects of VR, such as simulator sickness (Park et al., 2022), might act as a negative influence on learning. Overall, the effect of VR on learning success and experience is unclear. Nevertheless, VR training is used in many different industries (Andersen et al., 2020). We want to analyse its consequences in the context of intralogistics to ensure that it is a useful addition to existing training and does not decrease the already low motivation in logistics (Czernin and Schocke, 2016). Therefore, we not only focus on learning success, but also on other factors that can give an indication of the user’s learning experience. Mood is one of the factors that has been linked to learning (Olmos-Raya et al., 2018). Others include motivation (Gopalan et al., 2017), cognitive load (Kirschner, 2002) and flow (Oezhan and Kocadere, 2020).

In the next section we present the serious game InGo, which is the training at focus of this work. In the related works and research questions chapter we discuss previous research and derive our research questions from the research gaps. The methodology chapter explains our procedure and the used questionnaires. The results chapter provides both the demographic data as well as the analysis of the research questions. In the final chapter the results are discussed, we also consider the limitations of our work and give directions for future research. We end with the conclusion and our acknowledgements.

The VR based educational serious game InGo (Incoming Goods) simulates the intralogistics process of incoming goods. As a simulation, the serious game can overcome challenges of learning in the real-life situation, such as disturbing work processes or getting into dangerous situations. The game has been examined in a previous study in terms of physical and cognitive ergonomics, with positive findings regarding different aspects of the learning experience (Schlüter and Kretschmer, 2020). The study confirmed that InGo is a user-friendly learning method. It is intuitive, motivating and supportive. The study also evaluated the quality of the virtual environment itself: The participants had a good sense of presence in the virtual environment even though the experienced realism and involvement remained neutral. However, it should be be noted that this study did not compare the merits of InGo to traditional training methods, and the authors recommended further research in that particular direction. InGo includes a detailed description of the process to learn. In short terms, the participants learn the typical steps of the incoming goods process, i.e. checking and inspecting on the delivery address, delivery authorization, delivery time, parcel quantities, physical integrity of goods and transport packaging. In each step, the training can present a valid process or a deviation, e.g. a wrong address, and the users have to decide how to progress. They can identify the deviation and react accordingly, e.g. calling a supervisor and get further instructions on how to continue. The user or the trainer can configure the process and which deviation may occur. This ensures targeted training for each individual user.

The game aspect of the application is realized in various game elements like an overarching, decision-based narrative with different variations which is driven by the truck driver Ingo who has a delivery for the participants. InGo also acts as a trainer who is giving instructions on each process step (Tutoring System). Other examples for game elements are different interactive and animated elements and specific feedback for the player at the end of the game. An example for the interaction and animation in the game can be seen in figure 1.

The main goal of this work is comparing the effect of InGo on learning experience and success to traditional (e.g. paper based) learning.

Many different learning theories try to describe how humans learn. The four most dominant theories are the following: behaviourism, constructivism, communities of practise and connectivism (Campbell et al., 2020). Behaviourism supports different learning methodologies, such as practice, repetition and feedback, which enable memory associations. Still it leaves several questions unanswered. It does not focus on how the mind influences learning or why individuals do not learn equally when having the same teaching (Campbell et al., 2020). In communities of practice an activity is learned or improved through regular interactions (Wenger, 1999; Campbell et al., 2020). Researchers encourage the use of an authentic context to ensure learning, following this theory (Lave, 1988; Campbell et al., 2020). Following connectivist learning theories, learning is based on the construction and use of networks, which can be made up of humans and non-human information points (Campbell et al., 2020). Aspects of those learning theories are relevant for the learning approach taken in this research. However, constructivism might be the most fitting.

Following this learning theory, it is assumed that learning is the way we construct meaning from experiences (Scavarelli et al., 2021). In order to enable the construction of a meaningful and useful model of reality the learning process and the experiences should be realistic. Research proposes that it is important to enter as realistic situations as possible if the real-life situation is not available, for example, due to safety or monetary reasons (Scavarelli et al., 2021). The use of a VR serious game allows for overall realistic experiences and thus the construction of a useful model of reality, so it fits well into constructivist learning principles. Structurally, it follows basic principles of behaviourism, such as repetition and feedback. Feedback can be achieved directly through testing and evaluating (Schlüter and Kretschmer, 2020), which is difficult to achieve with, for example, paper based learning. It can also enable a more authentic context. Thus, it follows prinicples of communities of practise.

Overall, VR could be a facilitator for learning as its use follows different learning principles which in turn could improve learning compared to other learning methodologies. Different, especially constructivist learning theories seem to support VR as an improved learning approach. Past research also indicated that VR could act as an improvement compared to other, traditional learning methods. Previous works show a better performance for VR than for traditional teaching methods such as whiteboard, slides and projector (Ray and Deb, 2016). This improvement was also visible as performance in a real setting (Clark et al., 2016; Schlüter and Kretschmer, 2020). Still, participants will be less familiar with VR which can lead to the overall experience being more exhausting (Birbara et al., 2020). Overall, VR requires time to get used to, which can delay learning effects (Boyles, 2017). Overall, there are aspects of VR that support its advantages for learning, however other aspects could limit its benefits. Therefore we want to compare VR and traditional, paper based learning methods and find out how they differ in terms of the learning success. The first research question is:

RQ1: Is there a difference between the use of VR and paper based learning regarding learning success?

Different previous studies show that VR can reduce negative emotions and promote positive ones (Zhai et al., 2021; Chan et al., 2020). It could even reduce the effects of depression. Chan et al. had a similar focus to this current study. They compared different aspects of positive and negative moods between VR and pen-and-paper activities (Chan et al., 2020). They found that VR significantly improved positive moods and reduced negative moods. Furthermore, the reduction in negative moods was significantly greater for VR than for the pen-and-paper activities. Overall past works indicate the positive effects of VR on different aspects of mood. In the past mood has also been connected to learning and the effects of positive moods on improved learning could be shown (Olmos-Raya et al., 2018). Based on the found connections we want to analyse if and how all three factors are connected. The second research question is:

RQ2: What is the relationship between the use of VR, user mood and learning success?

In the past, prior knowledge could be identified as a moderating influence in different contexts (Kardes et al., 1994). Some research showed that prior knowledge and experience with a system positively influences the effectiveness of its use (Fathema and Akanda, 2020). However, other research found that different dimensions of prior experience influence future use differently (Varma and Marler, 2013). Concerning prior system knowledge and learning success different results exist as well. While some works show that learners who have more experience with the system are more satisfied with their course delivery medium, others did not find any differences in perceived learning and satisfaction (Wan and Fang, 2006; Marks et al., 2005; Arbaugh and Duray, 2002). In our third research question we want to find out, whether prior VR knowledge influences the relationship between learning method and learning success.

RQ3: Does prior VR knowledge moderate the relationship between the learning method and learning success?

Not only prior system expertise can be influential, but also domain knowledge has been shown to have an influence on learning effects (Johnson et al., 2015). Past works found prior knowledge as an important and significant predictor independent of the learning method (Beier and Ackerman, 2005; Kennedy et al., 2015). However, some research could also show that learners with a lower domain knowledge were able to achieve a bigger improvement than ones with a higher prior knowledge (Mitchell et al., 2005). The participants in this study will have prior knowledge concerning incoming goods processes as they are studying for their logistics degree. As it is unclear how differences in their prior knowledge might affect the relationship between learning method and knowledge test result, we ask the following in our fourth research question:

RQ4: Does prior knowledge on the topic moderate the relationship between the learning method and learning success?

One of the drawbacks of VR can be simulator sickness. It is one of the main reasons limiting VR adoption (Park et al., 2022). Different features of VR could be the reason for experiencing simulator sickness (Moss and Muth, 2011; Munafo et al., 2017). Overall simulator sickness is a phenomenon that is described in many different works, however the temporal patterns, its persistence after use and possibilities of adapting to it differ (Dużmańska et al., 2018). There have been signs that simulator sickness negatively influences performance (Yörük Açıkel et al., 2018). Little research has analysed the connection between simulator sickness and learning success, the research that did focus on this relationship did not find a connection (Johnson, 2005). We want to analyse the connections between learning method, simulator sickness and learning success to find out whether simulator sickness could be a challenge to the adoption of VR in learning contexts, similar to other contexts (Park et al., 2022). Thus our fifth research question is as follows:

RQ5: What is the relationship between learning method, simulator sickness and learning success?

Motivation is an important concept to explain human behaviour (Gopalan et al., 2017). Motivation is described as the route towards behaviour and the reason why individuals replicate behaviour. It allows humans to maintain goal-oriented behaviours. Motivation is of course an important aspect in learning (Gopalan et al., 2017). A positive connection between learner motivation and perceived learning could be shown in past research (Benbunan-Fich and Hiltz, 2003). Furthermore, high intrinsic motivation is associated with better learning results in long-term studies (Oezhan and Kocadere, 2020). In the context of learning VR and motivation have also been connected. Previous research found virtual learning environments to significantly increase motivation (Dalgarno and Lee, 2010; Shaw et al., 2017) in comparison to paper- or video based learning (Sattar et al., 2019). However, some research found contrasting results. They showed that higher intrinsic motivation does not lead to better learning results, instead it leads to different, more exploratory learning behaviors (Martens et al., 2004). As the three constructs have not been analysed together and even the connection between motivation and learning success is not completely clear, we want to find out how learning in VR, intrinsic motivation and learning success are related.

RQ6: What is the relationship between learning method, intrinsic motivation and learning success?

According to Cognitive Load Theory, a commonly used framework in learning and instruction (Armougum et al., 2019), there is a limited capacity to our mental resources at any given moment and the learning efficiency is strongly connected to the way we use these resources (Korbach et al., 2017). Cognitive Load Theory can be used to optimise information in a way to facilitate learning (Kirschner, 2002). Since VR offers a richer sensory experience in comparison to other media, there also are more stimuli to process, which could lead to a higher cognitive load (Albus et al., 2021). Past research found an increase in cognitive load through VR and especially highly immersive VR (Frederiksen et al., 2020). Further cognitive load could be added when individuals need to learn how to use a VR device in a learning situation (Schlüter and Kretschmer, 2020). Different approaches exist to solve this problem (Albus et al., 2021). However, some past works already suggest that VR can reduce cognitive load, for example by limiting cognitive load from external influences (Lee and Wong, 2014). As there are reasonings for VR both reducing or increasing cognitive load, we want to analyse this further and test the relationship between learning method, cognitive load and learning success.

RQ7: What is the relationship between learning method, task load index and learning success?

Flow is a mental state that is experienced when skills and challenge are approximately equal (Reid, 2004). A flow experience is perceived as totally satisfying (Clarke and Haworth, 1994). Individuals who experience flow tend to continue the activity simply because they enjoy it (Oezhan and Kocadere, 2020). Different design principles to foster flow exist (Kiili et al., 2012). VR has been shown to foster flow (Ruvimova et al., 2020), which in turn has been linked to learning (Oezhan and Kocadere, 2020). There also has been the indication that flow has a positive impact on game based learning outcomes (Perttula et al., 2017). Still, little research has so far focused on flow in serious games (Perttula et al., 2017). We want to analyse the relationship between learning method, flow and learning success empirically as suggested in past research (Perttula et al., 2017).

RQ8: What is the relationship between learning method, flow experience and learning success?

In this study we aimed to compare traditional paper based and VR enhanced learning methods and test them and other factors as influences on learning success. The base for the VR enhanced learning methods in this study was the serious game InGo which focuses on the logistics topic of handling incoming goods. The study was executed with students of a German vocational school that offers different courses in logistics. Two of their classes formed our respective groups. Both groups went through a process consisting of an initial questionnaire, an intervention and a final questionnaire, all completed within a single session. The entire study was conducted in German. The classes were randomly assigned to either the VR group or control group. Both groups took the survey in supervised group settings. They were greeted and informed about the process. Then they were asked to agree to conditions of the survey, which all of them did. Any prior questions could be answered during this part. Afterwards, the survey started with the initial questionnaire.

The initial questionnaire was structured as following: First we asked the students for their age and gender. Then we asked about simulator sickness with a questionnaire adapted from the simulator sickness questionnaire, which was measured on a four-point-scale from "not at all" to "strongly" (Tauscher et al., 2020). Afterwards we measured positive and negative mood aspects with a shortened PANAS five-point-scale (Mackinnon et al., 1999). The shortened PANAS consists of five items to measure positive and negative moods respectively. The values are added to get the score for either positive or negative mood. We also questioned the participants about their prior knowledge concerning both VR and incoming goods processes. For that we adapted two questions from previous research (Shou and Olney, 2020). The first question measures familiarity on a five-point-scale from "not at all familiar" to "very familiar". The second question measures experience with the following scale: "Never heard of it - Occasionally heard of it but have no personal experience - Frequently heard of it but have no personal experience - Have heard of it and have personal experience - Have frequent personal experience".

The intervention followed, during which both groups learned about the same topic. However they learned in different ways depending on their group: The VR group played InGo on Oculus Quest 2 VR devices provided by the school and the control group received traditional learning materials on paper. The VR group played through the steps of an incoming goods process included in InGo, such as checking and inspecting the delivery adress. The control group read through a paper based document, which also included the different steps. The document was based on InGo and partially course content provided by the school. It also included screenshots taken from InGo. The participants were asked to repeat and revise the content as often as they needed before proceeding. The intervention was executed in the same setting as the rest of the survey. The VR group received additional technical support from the research team in case of problems arising from the VR setup.

The second questionnaire followed after the intervention. Simulator sickness and mood were measured again. Next we measured intrinsic motivation with a translated KIM scale on a five-point-Likert-scale (Wilde et al., 2009). We measured the perceived task load with the translated NASA raw TLX (Moroney et al., 1992). The rTLX was measured on a scale from 0 to 100. The perceived flow was measured on a seven-point-Likert-scale with a questionnaire that was translated to German (Georgiou and Kyza, 2017). Lastly a multiple choice test based on the knowledge to be acquired during the intervention was used to measure the learning success, similarly to past research (Omelicheva and Avdeyeva, 2008). One example question would be: "Where are the packages taken for inspection?" with the following possible answers: "To the office of the plant manager", "To the shelf in the warehouse", "To the incoming goods area (at the loading dock)" or "To the outgoing goods area". For each correct answer participants gained a point. Multiple marked answers, own additions or no answers were considered as wrong answers. The points were afterwards added together to form the result.

Overall, we had a sample size of N = 41 participants. 21 participants were grouped in the VR group, while the other 20 were part of the control group. All participants were in school to pursue a logistics degree. No complete data sets had to be disregarded as no anomalies such as the same answers for many questions or unrealistic answers were observed.

The demographic data was analysed with SPSS 28.0.1. The VR group had 18 male and three female participants. The average age was M = 21.19, SD = 2.75. The age ranged between 16 and 27. The VR experience of the participants had an average of M = 2.65, SD = .81, while the VR familiarity had an average of M = 1.62, SD = 1.02. The average of the familiarity with incoming goods processes had an average of M = 3.1, SD = 1.18. The experience with incoming goods services had an average of M = 3.81, SD = .93.

The control group had 17 male and three female participants. The average age was M = 21.5, SD = 2.72. It ranged between 18 and 30. The VR experience had an average of M = 3.39, SD = 1.04. The VR familiarity experience had an average of M = 2.89, SD = 1.49. The familiarity with the incoming goods processes had an average of M = 4.06, SD = .94. The experience with incoming goods processes had an average of M = 4.28, SD = .75. There was no significant difference between the two groups concerning gender t(39) = -.063, p = .950 or age t(39) = -.362, p = .719.

However there were significant differences between the groups regarding VR experience t(39) = -2.458, p = .019, VR familiarity t(39) = -3.138, p = .003 and familiarity with incoming goods processes t(39) = -2.781, p = .008. Process experience showed no significant difference t(39) = -1.711, p =.095. The values for familiarity and experience of both groups can be seen in figure 2.

Several participants left out single items or whole scales. If participants left out single items the whole scale was disregarded to avoid bias (Mazza et al., 2015). The analysis of the research questions was conducted with SPSS 28.0.1. For parts of it the PROCESS macro by Hayes was used as well (Hayes, 2014). The macro uses ordinary least squares regression.

R1. There was no significant difference between the two groups concerning the learning success meaning their test results, t(39) = -1.446, p = .156. The mean values and standard deviations of the test results among the values of the mood variables can be seen in table 1. The paper based group achieved slightly higher scores in the knowledge test than the VR group.

Table 1. Mean and standard deviation values for learning success and mood

Group	Learning success		Positive mood before		Negative mood before		Positive mood after		Negative mood after
	M	SD	M	SD	M	SD	M	SD	M	SD
VR	7.38	2.36	13.48	4.56	7.00	2.58	15.19	5.19	6.05	2.38
Paper based	8.35	1.90	12.56	4.53	6.95	2.09	11.88	4.03	6.89	2.18

R2. Regarding the relationship between the use of VR, user mood and learning success, there were different aspects to be explored. As seen in table 1 positive moods were significantly enhanced after the use of VR in comparison to the mood before use from M = 13.48 to M = 15.19, t(20) = -4.683, p < .001. Negative moods were significantly reduced after the use of VR from M = 7.00 to M = 6.05, t(19) = 2.468, p = .011 (figure 3a). There was a significant difference between the two groups concerning the positive mood after the task, t(38) = 2.112, p = .042, with the VR group having an average of M = 15.19 and the paper based group with an average of M = 11.88. However, there was no significant difference between the two groups concerning negative moods after the task, t(38) = -1.170, p = .249 (figure 3b). Regarding the negative moods after the intervention the VR group had a slightly lower average of M = 6.05 than the paper based group with M = 6.89.

We then looked at the mediating influence of the mood regarding the two groups’ performance on the knowledge test. The requirements for a mediation analysis are linearity, normal distribution of the residuals, homoscedasticity, independency and temporal precedence (Hayes, 2014). The last two requirements are already given through the design of the study. Concerning the normal distribution and the homoscedasticity robust methods were chosen to compensate them if necessary. All variables had an approximately linear relationship, as assessed through the scatterplots after LOESS smoothing in a visual inspection.

As seen in figure 4 there was no effect observed from the learning method on the learning success, B = .6815, p = .364. However, newer research suggest this effect is not necessary for a mediation (Zhao et al., 2010). The two other paths were significant. The grouping predicted the positive mood significantly B = -3.3155, p = .040. The positive mood did predict the test result significantly B = .1796, p = .041. Still, we found only a marginally significant indirect effect, thus our results indicate that the relationship between learning method and learning success could be mediated by positive mood, indirect effect ab = -.5954, 95%-CI [-1.3178, .0206].

Concerning the mediation through negative mood there was no significant effect from the learning method on the learning success B = .9348, p = .189. The learning method did not significantly influence the negative mood,

B = .8471, p = .259. The negative mood also did not predict the learning success, B = -.1547, p = .303. The relationship between learning method and learning success was not mediated by negative mood, indirect effect ab = -.1311, 95%-CI [-.7779, .2246].

R3 Research question three focused on whether the relationship between the learning method and the learning success is moderated by the previous VR knowledge. The requirements for a moderation analysis are linearity, normal distribution of the residuals, homoscedasticity and independency. Independency is given through the design of the study. Concerning the normal distribution and the homoscedasticity robust methods were chosen to compensate them if necessary. The analysis showed a non-significant overall model F(3,34) = 2.25, p = .100, predicting 14.14% of the variance. Following our results, prior VR knowledge did not significantly moderate the relationship between learning method and learning success ∆R2 = 8%, F (1, 34) = .0175, p = .896, 95% − CI[−.8950, .7458].

R4. For the fourth research question the moderation of the relationship between the learning method and the learning success through the prior knowledge about incoming goods was tested. The analysis showed an almost significant overall model F(3,35) = 2.70, p = .060, predicting 39.49% of the variance. Yet prior incoming goods knowledge did not significantly moderate the relationship ∆R² = 0.3%. F (1, 35) = .2026, p = .655, 95%-CI[-.4348, .6751].

R5. The fifth research question considered the relationship between learning method, simulator sickness after the task and the learning success. There was no significant difference between the two groups with respect to simulator sickness, t(35) = -1.234, p = .225. The VR group experienced slightly less simulator sickness symptoms with an average of M = 1.26, than the paper based group with M = 1.38, (table 2, figure 5a).

Table 2. Mean and standard deviation values for simulator sickness, intrinsic motivation, task load and flow

Group	Simulator sickness before		Simulator sickness after		Intrinsic motivation		Task load index		Flow
	M	SD	M	SD	M	SD	M	SD	M	SD
VR	1.30	.30	1.26	.30	3.96	.46	34.42	13.02	4.65	2.18
Paper based	1.41	.37	1.38	.31	3.49	.73	40.99	15.00	3.35	1.12

Next we looked at a possible mediating influence of the simulator sickness regarding the learning method and the learning success. All variables had an approximately linear relationship, as assessed through the scatterplots after LOESS smoothing in a visual inspection. There was no significant effect from the learning method on the learning success, B = .6815, p = .364. As this is not necessary for a mediation the other paths were tested as well.

The learning method did not predict the simulator sickness significantly, B = .1239, p = .242. Simulator sickness also did not predict the learning success, B = -2.0727, p = .173. Thus the relationship between learning method and learning success was not mediated by simulator sickness, indirect effect ab = -.2568, 95%-CI [-1.0793, .2114].

R6. For research question six we focused on intrinsic motivation in different learning situations. The VR group had significantly more intrinsic motivation for their task than the paper based group, t(31) = 2.254, p = .016 (table 2, figure 5b). The VR group reached an average value of M = 3.96, while the paper based group had an average of M = 3.49.

We also looked at a possible mediating influence from intrinsic motivation on the relationship between learning method and learning success. There was no significant effect from the learning method on the learning success, B = .8222, p = .318. As this is not necessary for a mediation the other paths were tested as well. The learning method did predict the intrinsic motivation significantly B = -.4694, p = .045. The intrinsic motivation did not influence the learning success significantly B = 1.4735, p = .119. In our case the relationship between learning method and learning success was not mediated by intrinsic motivation, however the indirect effect was almost significant ab = -.6917, 95%-CI [-2.0213, -.0055].

R7. There was no significant difference between the two groups in the perceived task load, t(34) = -1.406, p= .169. The VR group perceived a slightly lower task load with an average of M = 34.42 than the control group with M = 40.99, (table 2, figure 5c).

We also tested whether the perceived task load index mediated the relationship between learning method and learning success. There was no significant effect from the learning method on the learning success, B = 1.0750, p = .171. As this is not necessary for a mediation the other paths were tested as well. The learning method did not significantly influence the perceived task load index, B = 6.5729, p = .188. Yet the perceived task load index did significantly predict the learning success, B = -.0534, p = .044. The relationship between learning method and learning success is not mediated by the perceived task load index, indirect effect ab = −.3510, 95%-CI [-.9248, .2118].

R8. The last dimension we focused on was flow. As seen in table 2 and figure 5d the VR group experienced significantly more flow with an average of M = 4.65 than the paper based group M = 3.35, t(29.309) = 2.326, p =.014.

The mediating effect of flow on the relationship between learning method and learning success was also examined. There was no significant effect from learning method on learning success, B = 1.1118, p = .146. The learning method did significantly predict flow, B = -1.2971, p = .030. Yet flow did not significantly predict the learning success, B = -.1939, p = .402. The relationship between learning method and learning success is not mediated by the perceived flow, indirect effect ab = .2515, 95%-CI [-.3065, 1.0310].

The two questions in focus of this work were: What is the impact of the use of VR regarding learning success when comparing it to traditional paper based learning? and What are other factors that play a role? For this we designed a study with two groups that would learn about an intralogistics topic either via means of a VR serious game or via text on paper.

Our first result was that there was no significant difference between the two groups concerning learning success. This indicates that VR and traditional paper based learning methods might be equally effective at conveying knowledge. It has to be noted though that the control group had a significantly higher prior familiarity with the topic than the VR group, which could have led to significantly higher results in the knowledge test in this group (Ledermüller and Fallmann, 2017; Johnson et al., 2015), but did not. This indicates a positive influence of the use of VR which might have had a compensating effect on this parameter meaning that discrepancies in prior knowledge might be easier to overcome with VR than with other learning methods. So it can be assumed that the VR group might have performed better than the paper based group if it had had the same prior knowledge on the topic. Past research could show the positive influence from prior knowledge on learning (Ozuru et al., 2009; Beier and Ackerman, 2005; Kennedy et al., 2015). In that regard one could argue that VR was even more efficient than the paper based group at conveying the information. Furthermore the VR group reached significantly positive results concerning moods, intrinsic motivation and flow, which are all facilitating factors in learning (Oezhan and Kocadere, 2020; Muilenburg and Berge, 2005; Bryan et al., 1996). Thus we can state that VR does have positive effects on the learning experience, even if no direct effect on the learning outcome was measurable in our study. These findings are in accord with previous studies on the learning experience with VR (Ray and Deb, 2016; Oezhan and Kocadere, 2020; Allcoat et al., 2021), which found that while VR may not improve learning success in the short-term, it certainly can in the long-term.

As mentioned before, there was a significant, positive effect of the use of VR on the participants’ mood. Since the mood was measured both before and after each intervention, it was possible to compare within and between groups. The results show that there has been an improvement in both positive and negative mood aspects within the VR group from before to after the intervention. This is in accord with past research (Zhai et al., 2021; Chan et al., 2020). Negative moods after the task were not significantly different between the groups, which contrasts with past research (Chan et al., 2020). However, the positive mood after the task was stronger in the VR group than in the paper based group. In the context of learning this is an important finding as mood is known to be an important predictor in long-term learning outcomes (Bryan et al., 1996; Nadler et al., 2010). In addition, we found an almost significant mediation effect between learning method, positive mood aspects and learning success. While the paths between the grouping and positive mood, and positive mood and learning success were significant, the indirect effect was almost significant. This might be explained by the relatively small sample size. Past research already showed the connection between mood and learning (Nadler et al., 2010). Fitting to past research, our data indicates the fact that positive mood aspects mediate the relationship between learning method and learning success, which can be extended to other forms of learning as well.

Since we found no significant moderating effect of VR familiarity on the relationship between learning method and the learning outcome, we cannot say for certain if it influences this relationship or not. However, if we assume a normal distribution, meaning a balanced number of experienced and inexperienced users in our VR group, and seeing that the VR group performed similarly well as the paper based group even though they were less familiar with the topic, this might indicate that this factor has no influence over the learning outcomes in this scenario. The use of VR in education seems not to be hindered by the experience level of the users. This can be explained as VR tends to be perceived as an intuitive technology. Past research could show that VR was able to minimise performance differences between novice and intermediate users (Le et al., 2020). However, we could not replicate a similar effect for learning. As our data was inconclusive in this case we would encourage further research in this matter.

There were significant differences between the two groups concerning the familiarity with incoming goods. These differences could not be prevented as the groups were assigned randomly. However, a more similar familiarity basis might have led to a better comparability of the two groups. We expected that the two groups’ unequal preconditions in these aspects would influence their performance in the knowledge test since prior knowledge is a well-known predictor of learning success (Johnson et al., 2015). However, there were no significant differences with regards to learning outcome. This leaves to be assumed that the VR group might have performed even better than the paper based group if it had had the same prior knowledge level on the topic. Groups with a similar basis concerning knowledge and familiarity should be ensured in future research.

We found no significant difference between the VR and the paper based group with respect to simulator sickness. The paper based group even experienced slightly stronger symptoms than the VR group. This could be explained by the short duration of the intervention (Johnson, 2005) and the high quality of the hardware used in the VR group (the Oculus Quest series shows high levels of comfort (Grassini and Laumann, 2021)), as well as an adequate level of software optimization. The VR group experienced significantly higher positive moods which might also influence the perception of negative simulator sickness symptoms. Two further factors influencing this issue might be derived from the study setting: The groups were taught independently on two different days. While the VR group had a rather unfamiliar and disrupting study setting, which might have been perceived as more exciting compared to the study setting of the paper-based group, which was very similar to their daily school routine. Additionally, due to different schedules the VR group was tested on a Friday, while the paper based group was tested on a Monday, which also might have affected their moods and health status. Coming back to the context of learning, there is little evidence that would connect higher simulator sickness values to lower learning outcomes (Johnson, 2005), even though it may affect the performance while using the simulator (Yörük Açıkel et al., 2018). This indicates that simulator sickness may not significantly hinder the adoption of VR for learning purposes.

We observed a significant impact of learning method on intrinsic motivation. The use of VR is positively connected to the intrinsic motivation of the players. The result is consistent with past research (Dalgarno and Lee, 2010; Sattar et al., 2019). Motivation is viewed as the reason why humans repeat behaviour (Gopalan et al., 2017). Our results could thus indicate that students would be more willing to continue learning via VR, which, following past research, would improve learning results (Oezhan and Kocadere, 2020). However, we could not confirm a mediating influence of intrinsic motivation onto the relationship between learning method and learning success. We did however find a significant relationship between learning method and intrinsic motivation, both the relationship between intrinsic motivation and the indirect effect were marginally significant. While not significant our results indicate a probability of a mediation between the three factors. This inconclusive result could be related to the relatively short length of the intervention. Motivational factors have been known to influence the learning outcome in the long-term (Oezhan and Kocadere, 2020; Muilenburg and Berge, 2005). Besides, there is evidence for a positive influence of motivational aspects onto flow, which is known to positively influence learning outcomes (Oezhan and Kocadere, 2020). Thus, an increased motivation could be interpreted as a positive influence on the overall learning experience with a potential for long-term effects on the learning outcome.

According to Cognitive Load Theory, a commonly used framework on learning and instruction, there is a limited capacity to our mental resources at any given moment and the efficiency of knowledge acquisition is strongly connected to the way we use these resources (Korbach et al., 2017). Cognitive load was measured using the perceived task load index. Since the VR group did not differ significantly from the control group with regards to cognitive load and the design and control of the VR serious game was more complex and unfamiliar than the paper based method in this study, this could indicate that the VR serious game accounted for less cognitive load than the paper based learning materials. Some past research has also indicated this (Lee and Wong, 2014). This shows potential for further improvement of VR applications: if VR familiarity increases and the application design improves, this could lead to a decrease in cognitive load. Our data suggests that a decrease in perceived task load has a positive influence on learning outcomes.

Flow was one of the aspects that was significantly improved by the use of VR, supporting past research (Ruvimova et al., 2020). However, it did not predict the learning success. This does not confirm findings from other sources that suggest a strong interconnection between flow and motivational aspects (Rheinberg and Engeser, 2018) which in turn have a strong positive influence on long-term learning success (Oezhan and Kocadere, 2020; Muilenburg and Berge, 2005). Similarly to the motivational aspect, this might be explained by the short time of the intervention as well as the directly subsequent testing which allowed for assessment of short-term learning effects only. Furthermore, it has been suggested to measure flow in a continuous manner (Perttula et al., 2017) rather than using a one-time-assessment method at the end of the intervention. However, it can be deduced that a significant increase in flow itself indicates an improvement of the learning experience (Clarke and Haworth, 1994; Yeh et al., 2019). In turn this could lead to students being more willing to repeat the task, similarly to motivation (Gopalan et al., 2017), which could lead to long term learning effects.

A major limitation was caused by the group setting of the studies. Participants may have spoken to each other despite being instructed not to do so, which may have influenced their answers in the questionnaires. A study design based on single participants would improve on this issue and enable us to diversify the groups to minimise differences regarding prior knowledge among others.

Also, the groups were predetermined by the school, meaning that all participants from one group belonged to the same class and shared very similar prior domain knowledge due to their shared schedule. In future research the participants should be grouped in a randomized fashion to prevent prior differences and in turn show differences induced by the experimental design more clearly. This might show the differences between the two learning methods more clearly, similarly to past research (Ledermüller and Fallmann, 2017; Johnson et al., 2015).

Another factor to consider was the structural similarity between the paper based learning materials and the paper based test method, which might have further amplified the advantage the paper based group had over the VR group. In order to make the test equally challenging to both groups it might be a good approach for the future to design the test to be very different from both interventions. In our case this might have been achieved by doing a practice exam, which would have required a single-participant design. Furthermore, it has to be noted that the students were used to paper based learning, but they had not used VR as a learning method in school. As VR takes time to get used to, different results could possibly be obtained in a longer study (Boyles, 2017); however, this wasn’t possible in the given organizational context at a public school with limited resources regarding staff, time and available rooms.

Moreover several participants did not fill out whole questionnaires and therefore reduced the amount of usable data. To prevent this in the future more participants could be recruited, additional approaches for promoting conscientiousness in participants could be implemented in the study design or an online questionnaire could be used that does not allow leaving questions blank as easily. In future research should aim to recruit more participants, not only to be able to handle missing data, but also to ensure reliable results. In logistics recruiting participants can be difficult, which might be caused by their low motivation for their work (Czernin and Schocke, 2016). Thus, we decided to work with a school that already used our trainings. However, they also could not let too many students miss class, thus limiting our sample.

Another major limitation of this study was that the knowledge test was filled out immediately after the intervention, uncovering only short-term learning effects. In order to asses the whole range of effects of VR based learning however, it is important to assess long term effects as well. Past research indicates that VR can be helpful in creating virtual "memory spaces" and thus facilitating memory recollection (Scavarelli et al., 2021). VR could be helpful in memorising the learned knowledge long-term, especially because it has shown to improve aspects like moods, flow and intrinsic motivation in our study, which are known to have effects on long-term learning success (Oezhan and Kocadere, 2020; Muilenburg and Berge, 2005; Bryan et al., 1996). Also as VR is rather uncommon in schools and takes time to get used to, different results could possibly be obtained in a longer study with repeated interventions (Schlüter and Kretschmer, 2020).

The last important aspect is the cognitive load, which was measured using the task load index (NASA rTLX). In this study, it was measured using the subjective rating scale introduced by the NASA Task Load Index (NASA TLX) post-intervention. This scale is commonly used for the measuring of cognitive load (Korbach et al., 2017). However, it is limited to a specific point in time. This leads to the problem that we have only selectively summarized and evaluated information about a matter that is continuously changing in the course of the learning process. This also applies to other aspects like mood or flow. There are several ways in which these aspects could be, at least in part, measured continuously, e.g. through the use of eye tracking technology (Korbach et al., 2017), video based approaches (Kannegieser et al., 2020) or different physiological measurements like skin conductivity, heart monitoring or neuro-imaging (Tyng et al., 2017). This might be valuable for further research as it might help better understand the structural differences between VR and other methods in education when it comes to these aspects, which are, as our research suggests, key factors in learning success.

The focus of this study was to examine the different effects of the use of VR in a learning environment. Specifically, we focused on its effects on learning success, motivation, mood and flow. For this we devised a learning study with two groups that would learn about a logistics topic either via means of a VR serious game or traditionally via text on paper. We discovered that both learning methods were equally effective at conveying knowledge. However there were several other aspects at which VR seemed to be more effective than paper, namely at improving mood, intrinsic motivation and flow. We also looked at possible moderating or mediating effects caused by one of these improvements, but could not confirm any. In this aspect it appears that our findings contradict previous findings in this field (Oezhan and Kocadere, 2020; Muilenburg and Berge, 2005; Bryan et al., 1996). One possible explanation for this might lie in different aspects of the study design that might have reduced the positive effects of the improved intrinsic motivation and flow of the participants as discussed in section 6.1. At around 15 minutes the intervention might have been too short to form an effect based on flow or motivational aspects alone. Also, the setting of the study allowed for several participants to be close by at any given moment, which could have caused distractions and diminished the flow. Another possibility might be that the paper based group was privileged in comparison to the VR based group in the sense that it had significantly higher prior knowledge, which in turn balanced out any advantages the VR group might have had.

In addition, we should point out that there was a higher similarity between the structure of the intervention and the knowledge test due to the test being in written form for the paper based group, which might have been to their advantage. In the future, this could be addressed by choosing a test design that is distinct from both interventions, such as a practical test. Though one might argue that this would be structurally closer to the VR experience and thus favor this group in turn.

Furthermore, the group situation of the study might have been the cause of different social effects. These might have affected the mood, motivation and even the knowledge of the participants. For future studies, we would suggest to favor a single-participant design over a group design if possible.

However, we found evidence for a positive impact of VR on user mood, flow and intrinsic motivation. Given these positive side effects and the findings of previous studies, we can safely say that VR has proven at least as effective as paper based methods at conveying practical knowledge, if not more so. These positive effects might be useful for teaching material that is difficult to convey in a traditional setting, like safety measures or the training of correct behaviour in dangerous situations.

Considering the limitations mentioned before as well as the results pointing towards equal effectiveness with positive side effects of VR, this clearly indicates a strong potential of this learning method in the context of education. Furthermore, the decreasing cost of VR equipment, along with the rising familiarity with VR as a technology and the spread of different educational tools and services for this platform, might make VR more accessible to educational institutions. This could inspire more educators and institutions to expand their teaching portfolio and include VR in their methodology.

We would like to extend our thanks to the students and teachers at the Berufskolleg Wirtschaft und Verwaltung Remscheid. This study was carried out as a project within the interdisciplinary Center of Excellence Logistics and IT¹

Boyles, B., 2017. Virtual reality and augmented reality in education. Center for Teaching Excellence, United States Military Academy, West Point, Ny.

Czernin, J. & Schocke, K.-O., 2016. Handlungsfelder der Personalarbeit in der Logistik. Studienbericht, Frankfurt University of Applied Science.

Fathema, N. & Akanda, M. H., 2020. Effects of instructors’ academic disciplines and prior experience with learning management systems: A study about the use of Canvas. Australasian Journal of Educational Technology, 36(4), 113–125. https://doi.org/10.14742/ajet.5660

Hayes, A. F., 2014. Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression- Based Approach. New York, NY: The Guilford Press. Journal of Educational Measurement, 51(3), 335–337. https://doi.org/10.1111/jedm.12050

Kirschner, P. A., 2002. Cognitive load theory: Implications of cognitive load theory on the design of learning.

Lamb, R. L., 2016. Examination of the Effects of Dimensionality on Cognitive Processing in Science: A Computational Modeling Experiment Comparing Online Laboratory Simulations and Serious Educational Games. Journal of Science Education and Technology, 25(1), 1–15. https://doi.org/10.1007/s10956-015-9587-z

Moroney, W., Biers, D., Eggemeier, F., & Mitchell, J., 1992. A comparison of two scoring procedures with the NASA task load index in a simulated flight task. In Proceedings of the IEEE 1992 National Aerospace and Electronics Conference@m_NAECON 1992, pp. 734–740 vol. 2.

Oezhan, S. C. & Kocadere, S. A., 2020. The Effects of Flow, Emotional Engagement, and Motivation on Success in a Gamified Online Learning Environment. Journal of Educational Computing Research, 57(8), 2006–2031. doi:10.1177/0735633118823159.

Ray, A. B. & Deb, S., 2016. Smartphone Based Virtual Reality Systems in Classroom Teaching — A Study on the Effects of Learning Outcome. In 2016 IEEE Eighth International Conference on Technology for Education (T4E), pp. 68–71. https://doi.org/10.1109/T4E.2016.022

Wan, Z. & Fang, Y., 2006. The role of information technology in technology-mediated learning: A review of the past for the future. AMCIS 2006 Proceedings, 253.